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Abstract 


We assessed the relationship of level of QualityCore® implementation and student 
achievement in Algebra I. The study is based on 1,291 9th grade students in three large 
metropolitan school districts who took EXPLORE in the fall and the QualityCore Algebra I end- 
of-course assessment (EOC) in the spring. During the same period, the students’ 41 Algebra I 
teachers completed questionnaires measuring their use of QualityCore’s tools and strategies for 
Algebra I. We found that level of teachers’ QualityCore implementation had a positive and 
statistically significant relationship with student achievement, measured by performance on the 
QualityCore Algebra I EOC controlling for performance on the fall EXPLORE tests. When 
different domains of implementation were considered, the Teaching Practices and Collaboration 
with Colleagues domains had the strongest relationship with student achievement. Survey results 
also indicate how QualityCore use varied across components and that QualityCore was well- 
received by teachers. 


IV 




Examining the Relationship of QualityCore® Implementation and Student Achievement in 

Algebra I 

Introduction 

Years of ACT research show that it is not the number of courses a student takes in high 
school, but what happens in those courses that matters most (ACT, 2005; ACT & The Education 
Trust, 2005). Despite educational reforms and improvement initiatives, the percentage of 
students ready for all four college courses has only increased slightly over the last ten years. In 
2011, only one in four students had at least a 50% chance of earning a B or higher college grade 
in all four first-year courses at a typical postsecondary institution: English Composition, College 
Algebra, Social Sciences, and Biology (ACT, 2011). 

In response to this lack of college readiness, ACT’s research and development team 
partnered with the nation’s leading educators to develop QualityCore as a tool for raising the 
quality and intensity of core high school courses (ACT & The Education Trust, 2005). 
QualityCore currently supports twelve core high school courses: English 9, 10, 11, and 12; 
Algebra I, Geometry, Algebra II, Precalculus, Biology, Chemistry, Physics, and U.S. History. 
QualityCore resources include the following components: 

• Instructional resources consist of a combination of course standards, test blueprints, 
and model instructional units that help educators customize instruction to the needs of 
their students. 

• Formative item pools and test builders are pools of formative items that can be used 
to create customized quizzes and temporal benchmark assessments. The use of the 
formative item pools saves time and provides timely feedback to students. 
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• Professional development resources and workshops help educators expand their 
teaching skills and learn about effective practices of other teachers. 

• End-of-course assessments (EOCs) with constructed response and multiple-choice 
options help educators evaluate student achievement gains in each course. 

• Score reports and progress reporting provide comparison of students’ performance 
on the EOCs at the local, state, and national levels. They also include evaluation of 
students’ progress towards college readiness in a given course. 

With the introduction of any school improvement initiative or new instructional program, 
school leaders and policymakers want to know if progress is being made in raising student 
achievement. Moreover, school leaders want to know what aspects of the program are working 
or not working, and how much teachers are utilizing the program. In response, this study 
examined the relationship between the levels of teachers’ implementation of QualityCore 
Algebra I and student achievement (measured by the QualityCore Algebra I EOC) for 9th grade 
students. A hypothetical example of a teacher with a high level of implementation is a teacher 
who 

• worked extensively with his or her colleagues to modify course objectives and lesson 
plans to address QualityCore course standards; 

• adopted and regularly used QualityCore resources that provide systematic approaches 
to examining student work and modifying lessons to make them more rigorous; 

• regularly used the QualityCore formative item pool for interim assessments and 
classroom instruction; and 

• evaluated his or her student’s performance on the EOC and used the results to set 
goals for next year’s students. 
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The study was conducted in three large metropolitan school districts with high 
concentrations of lower-achieving students, many living in poverty. Many of the schools were 
designated as needing improvement for not making adequate yearly progress (AYP) towards 
proficiency, as mandated by the No Child Left Behind Act (NCLB). 

The 9th grade Algebra I teachers underwent professional development (PD) to introduce 
them to the resources offered by QualityCore and how to integrate the resources into their 
instruction and collaboration. Most of the training took place in the summer before the 2009- 
2010 academic year. Training topics included defining rigor and relevance of high school 
courses, analyzing depth of knowledge of test items, analyzing course objectives to ensure 
alignment, research-based instructional strategies, developing quality in-course assessments, 
using constructed response items, and creating a course syllabus. Follow-up training was offered 
during the school year to reinforce the summer training and offer additional support for problems 
encountered throughout the school year. Topics included techniques for scaffolding instruction to 
match instruction, examining student work, and revising lesson plans. The last day of follow-up 
training was only available through online training in two of the districts; in the other district 
teachers could participate either online or through the traditional face-to-face training. 

In one district, all Algebra I teachers were invited to the PD; in the other two districts 
only selected teachers underwent training. (We did not collect information on how teachers were 
selected for training). During the same academic year, a sample of Algebra I teachers completed 
questionnaires about their implementation of QualityCore’s instructional practices, tools, and 
strategies. Although the districts implemented other QualityCore courses (English 9 and 
Biology), we were able to collect implementation data only on Algebra I. 
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Research Questions Addressed 

The primary research question addressed by the study is: 

1) To what extent did level of implementation of QualityCore Algebra I correspond to 
student achievement in Algebra I? 

Secondary research questions include: 

2) Which aspects of QualityCore implementation (domains) were most related to student 
achievement in Algebra I? 

3) Which components of QualityCore did teachers use the most? 

Methodology 

We used a longitudinal study design with measurements of QualityCore implementation 
occurring between students’ initial measure of academic achievement (EXPLORE) and students’ 
end-of-course measure of achievement. The study was carried out by administering surveys to 
teachers to measure their level of QualityCore implementation. The survey response data was 
then merged to their students’ test scores from fall 2009 (EXPLORE) and spring 2010 
(QualityCore). 

Measures of QualityCore Implementation 

Survey construction and administration. A survey was constructed by ACT staff in fall 
2009 to measure different aspects of teachers’ use of QualityCore resources. The survey content 
was based on QualityCore training materials, with a focus on the materials used for the 
professional development offered to teachers in the three large school districts where the study 
took place. The survey items covered the use of specific QualityCore tools and resources (e.g., 
the Eormative Item Pool, the Model Instructional Unit, the Template to Examine Assignments 
for Rigor & Relevance), as well as teaching practices that were not specific to QualityCore but 
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that targeted the professional development and/or are reinforeed by the QualityCore program 
(e.g., meeting with colleagues to review student work; designing a course syllabus; classroom 
routines such as summarizing lessons, and providing feedback to students). 

The first survey was administered in November 2009 and the second survey was 
administered in May 2010. In the fall, teachers were invited to participate in the survey using 
emails with links to the web-based survey. In the spring, paper and pencil versions of the survey 
were delivered in person to teachers with postage-paid return envelopes in an effort to improve 
survey response rates. The contents of the fall and spring surveys were not identical, but 
common domains of QualityCore implementation were measured by the two surveys. The fall 
survey included 49 items and covered more items related to teaching practices and collaboration 
that were less directly related to the QualityCore training objectives. We shortened the spring 
survey to encourage more survey respondents. The spring survey included 21 items and was 
focused more on use of QualityCore-specific resources. 

In all, 28 teachers completed the fall survey, 41 teachers completed the spring survey, 
and 13 teachers completed both the fall and spring versions. Thus, 56 teachers completed at least 
one survey, and we estimate that this represents 37% of all teachers in the three districts who 
implemented QualityCore Algebra I in 2009-2010. (An exact count of teachers who 
implemented QualityCore Algebra I was not available from the districts, but we estimated the 
count at 152 based on PD training roster data and information provided by some schools.) 
Among these 56, 41 teachers had students that took EXPLORE in the fall and the QualityCore 
Algebra I EOC in the spring. The 41 participating teachers were from 19 high schools in 3 large 
school districts. The sample contained 1,291 grade 9 students (with EXPLORE scores, 
QualityCore scores, and with teachers who took one or more implementation surveys); on 
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average, there were about 32 students per teaeher. This is not an estimate of average class size, 
however, because teachers could teach multiple sections of Algebra 1. Table 1 summarizes the 
samples of teachers and students with EXPLORE scores used in the study. The primary analyses 
are based on the sample of teachers and students with full data; secondary analyses use the entire 
sample of surveyed teachers. 

Table 1 
Study Samples 


Timing of 

implementation survey 

Data 

available 

N 

schools 

N 

teachers 

N 

students 

Eall 2009 

Total 

19 

28 


With student data 

16 

18 

507 

Spring 2010 

Total 

17 

41 


With student data 

17 

35 

1,117 

Combined 
Eall & Spring 

Total 

26 

56 


With student data 

19 

41 

1,291 


Implementation domains. The survey items were classified into one of five domains: 
Collaboration with Colleagues, Educator Resources, Formative Items, Teaching Practices, and 
QualityCore Adoption. The domain classifications were based on descriptions of QualityCore 
professional development (ACT, 2011) and were created with the goal of distinctively and 
exhaustively capturing the various aspects of implementation. Below, we describe the domains 
along with their connections to the overall goals of raising the intensity and quality of Algebra I 
course. The survey items under each domain for the fall and spring surveys are presented in 
Appendix A. 

Collaboration with Colleagues. This domain measured the amount of time teachers spent 
with colleagues examining student work, planning lessons, executing instructional strategies, and 
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evaluating student progress. It measured the effort and time teaehers spend working together 
towards developing Algebra I instructional strategies. 

Educator Resources. Educator Resources measured the use of specific QualityCore 
resources such as the template to examine assignments for rigor and relevance, depth of 
knowledge analysis, and worksheets to examine student work and determine expectations for 
high quality performance. In addition, it measured the use of teaching strategies from 
QualityCore's Educator’s Toolbox. This domain assessed the extent that teachers learned to use 
QualityCore Algebra I materials to reflect on, augment, and enhance their own instructional 
materials. 

Formative Items. The QualityCore professional development was intended to improve a 
teacher’s ability to “develop the capacity to analyze formative and summative test items for 
depth of knowledge and compare those items to differentiate level of (student) thinking required” 
(ACT, 2011). The Eormative Items domain focused on the use of formative constructed-response 
and multiple choice items from the QualityCore Eormative Item Pool (EIP). This domain 
assessed the extent that teachers used the PIP throughout the school year. 

Teaching Practices. Teaching Practices covered a variety of classroom and other 
practices supported by QualityCore training, model instructional units, and course syllabi. It 
included items related to classroom routines and organization, as well as some activities teachers 
performed outside of the classroom in preparation for lessons. 

QualityCore Adoption. QualityCore Adoption measured the extent that teachers and their 
colleagues had “bought in” to QualityCore and had ingrained QualityCore into their teaching and 
planning activities. It surveyed the level of familiarity with and the use of QualityCore 
instructional resources, such as model instructional units, as well as the extent that existing 



course standards were compared with QualityCore eourse objectives. This domain also measured 
expectations of and enthusiasm for the QualityCore program. 

Survey items and domain scoring. The survey items measured agreement, frequency of 
behaviors, or time spent. The three sets of survey response options were: 

• Agreement: NA = Not Applicable, 1 = Strongly Disagree, 2 = Moderately Disagree, 
3 = Slightly Disagree, 4 = Slightly Agree, 5 = Moderately Agree, and 6 = Strongly 
Agree. 

• Frequency of behaviors: NA = Not Applieable, 0 = Never (None), 1 = Seldom, 
2 = Sometimes, 3 = Often, 4 = Frequently, and 5 = Daily. 

• Time spent: 0 = No Time, 1 = One Hour, 2 = Two Hours, 3 = Three Hours, 4 = Four 
Hours, 5 = Five Hours, 6 = Six to Ten Hours, and 7 = Eleven or More Hours. 

Item responses were assigned numerie values corresponding to the order of the response options. 
Items with responses of “Not applicable (NA)” were assigned a missing value, while those with 
“Never” and “None” were assigned a 0. Although most of the prompts were positively framed 
(e.g., ""I plan on using this year’s QualityCore end-of-course exam results to set a baseline for 
future years ’ student achievemenC), some items, like, “/ don ’t have the time right now to use 
QualityCore elements in my work, ” were negative. Responses to items that were negatively 
structured were reverse-scored. 

Using the fall and spring survey data separately, we eomputed QualityCore 
implementation domain scores by taking the average of the items classified under each domain, 
resulting in ten possible seores for each teacher. Eaeh of the ten scores was then standardized to 
have mean 0 and standard deviation 1 . Then, we ealculated overall seores for the five domains 
by taking the average of eaeh pair of standardized spring and fall scores. The aggregate 
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implementation seore (eombining fall and spring and all domains) was defined as the average of 
the ten fall and spring domain seores. Overall fall (spring) implementation seores were obtained 
by averaging the five fall (spring) domain seores. By using standardized seores, we foreed the 
domain seores to earry equal weights in determining the overall measures of QualityCore 
implementation. For teaehers who only eompleted the fall (spring) survey, the eombined 
implementation measures (refleeting implementation throughout the year) were set to equal the 
standardized fall (spring) implementation eomponent. The resulting data included five scores 
measuring domains of implementation (Collaboration with Colleagues, Educator Resources, 
Formative Items, Teaching Practices, and QualityCore Adoption) and three aggregate measures 
of implementation - fall, spring, and combined (fall and spring). 

The aggregate variables measured the overall level of QualityCore Algebra I 
implementation, without regard to the various aspects (domains) of implementation. Unlike the 
domain-specific measures outlined above, the aggregate measures did not convey knowledge of 
what aspects of implementation were associated with changes in student achievements. 
However, the aggregate measures are believed to be more reliable because they are based on 
more survey items. 

Measures of Academic Achievement 

Fall EXPLORE test. The EXPLORE tests of educational development were 
administered to the 9th graders early in the fall. EXPLORE measures student development in the 
curriculum areas of English, mathematics, reading, and science. EXPLORE is most commonly 
used for students in 8th and 9th grades and focuses on knowledge and skills usually attained by 
8th grade. The EXPLORE scale scores in the four subject areas, which are reported on a scale 
from 1 to 25, were used as measures of initial academic achievement level. The reliabilities of 
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the EXPLORE scale scores are 0.85-0.87 (English), 0.84 (mathematics), 0.83-0.86 (reading), and 
0.79-0.84 (science) (ACT, 2007). These values give the degree of consistency in the test scores 
(ACT, 2007). Reliabilities closer to 1.0 indicates greater consistency or lesser error in test 
measurements while a value closer to zero reflect little or no consistency, or a higher potential 
for error in the test scores. 

Spring QualityCore test. The QualityCore Algebra I EOC was administered to the 9th 
graders in the spring. The assessment was aligned to ACT Course Standards (ACT & The 
Education Trust, 2005) and included problem-based items embedded in contexts that were 
accessible and relevant to high school students. Scores on the QualityCore Algebra I EOC range 
from 125 to 175, with reliability estimated at 0.75 (ACT, 2010a). The EOC scores were used as 
measures of end of course academic achievement level. 

Table 2 lists the variables used for analysis, classified according to the level — student or 


teacher — and when the data was collected. 
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Table 2 

Analysis Variables 


Variable 

Range 

Level 

Data Collection 

EXPLORE scores 




EXPLORE English 

1-25 

Student 

Pall 

EXPLORE Mathematics 

1-25 

Student 

Pall 

EXPLORE Reading 

1-25 

Student 

Pall 

EXPLORE Science 

1-25 

Student 

Pall 

QualityCore Algebra I score 

125-175 

Student 

Spring 

Implementation scores 




Eall Implementation 

1.94-5.13 

Teacher 

Pall 

Spring Implementation 

2.04-5.06 

Teacher 

Spring 

Aggregate Implementation 

2.04-5.10 

Teacher 

Pall and/or Spring 

Collaboration with Colleagues 

0.50-5.50 

Teacher 

Pall and/or Spring 

Educator Resources 

1.00-5.54 

Teacher 

Pall and/or Spring 

Eormative Items 

0.50-5.50 

Teacher 

Pall and/or Spring 

Teaching Practices 

2.20-5.57 

Teacher 

Pall and/or Spring 

QualityCore Adoption 

1.33-6.00 

Teacher 

Pall and/or Spring 

Note: Implementation score ranges bei 

bre standardization are given. 


Statistical Modeling 

Because students taking Algebra I were nested within teachers, we used a hierarchical 
linear model that accounted for unobserved teacher effects on student achievement (Goldstein, 
2002; Raudenbush & Bryk, 2002; Snidjers & Bosker, 1999). Because we anticipated unobserved 
teacher effects on student achievement, we specified a random intercept model. A significant 
variation in teacher intercepts (p-value < 0.05) would indicate variation in student achievement 
across teachers that is not fully explained by prior achievement and teachers’ QualityCore 
implementation. The hierarchical linear model was used to test the relationship between the 
different measures of implementation and student achievement. 

Student achievement was measured by QualityCore Algebra I EOC score, controlling for 
students’ initial achievement level, as measured by the four EXPLORE scale scores. By 
regressing the EOC score on measures of prior academic achievement (the four EXPLORE 
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subject test scores), other independent variables that are predictive can be regarded as 
contributing to student achievement. We included the group mean EXPLORE Mathematics score 
for each teacher as an additional covariate to capture possible peer effects (Angrist & Lang, 
2002): Students surrounded by higher- achieving peers tend to show greater academic 
achievement. Although the analysis included EXPLORE English, Reading, and Science, only the 
group mean EXPLORE Mathematics was used as a teacher-level covariate to capture each 
group’s prior mean mathematics level, which is most applicable to the dependent variable 
(QualityCore Algebra I score). Mean EXPLORE scores in the other subject areas were not 
included as predictors because we did not think they would help explain the variation in the 
dependent variable. 

To test the primary research question (“To what extent did level of implementation of 
QualityCore Algebra I correspond to student achievement in Algebra I?”), the aggregate measure 
of implementation was used as an independent variable in the hierarchical linear model. The 
primary analysis measured the relationship of QualityCore Algebra I level of implementation to 
student achievement. Other analyses tested for effects of different aspects of implementation, as 
measured by the domain implementation scores. 

Eor the ease of computation and interpretation, all predictors were grand-mean centered 
(Koenig and Lissitz, 2001) before inclusion in the model. Initially, we fit a model with the 
aggregate (combining fall and spring) implementation measure. Then we fit separate models for 
the fall and spring implementation measures. Next, we fit models using the five domain 
measures of implementation. Each domain measure was entered into the model individually, and 
then a joint model with all five measures was tested. Independent variables were considered 
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statistically significant if the parameter estimate signifieance test resulted in a p-value of less 
than 0.05. 

Summarizing Survey Data 

To address the third research question {Which components of QualityCore did teachers 
use the mostl), we assessed the survey response data from all responding teachers, including 
some that were not ineluded in the regression analyses because their students did not take 
EXPLORE or the EOC assessment. To summarize the survey data, item response data were 
eoded to numerie values eorresponding to the order of the response options (e.g., l=strongly 
disagree, 2=moderately disagree, etc.). We calculated eaeh item’s mean and standard deviation. 

Eor the items that used the agreement scale, we also reported the pereent of teaehers 
agreeing with the statement (slightly agree to strongly agree). Eor items that asked teaehers to 
report the number of days (out of the last 5 regular instruetional days) that they performed a 
eertain aetivity, we reported the pereentage of teaehers reporting 3 or more days of performing 
the aetivity. Eor items that asked teaehers to report the number of hours (during the last 5 regular 
instructional days) that they performed a certain activity, we reported the pereentage of teachers 
reporting 3 or more hours of performing the aetivity. 

Results 

The demographic composition of the sample is given in Table 3. The vast majority of 
students in the study sample were African American/Black (85%); Hispanic (6.9%), Asian 
(3.0%), and Caucasian/White students (2.6%) were less represented. Compared to the population 
of students enrolled in elementary and seeondary sehools in the U.S., the sample had a mueh 
larger concentration of Afriean Ameriean students, and a mueh smaller concentration of 
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Caucasian/White and Hispanic students. The sample was nearly evenly split by gender (51% 
female and 49% male). 

Table 3 

Student Demographics 


Ethnicity N (%) Population 

Race/Ethnieity 


Afriean American/Black 

1,096 (85.0) 

16.0 

Ameriean Indian, Alaskan Native 

11 (0.8) 

1.4 

Asian 

38 (3.0) 

4.4 

Caucasian/White 

34 (2.6) 

57.8 

Hispanie 

89 (6.9) 

20.4 

Multiraeial 

16(1.2) 

- 

Other 

1 (0.1) 

- 

Prefer Not To Respond/Missing 

4(0.3) 

- 


Gender 


Male 

632 (49.0) 

50.2 

Female 

657 (51.0) 

49.8 

Total' 

1,289 (100.0) 

100 


Due to rounding, sums may not equal 100. 

^Source: U.S. Department of Education, National Center for Education Statistics, 

Schools and Staffing Survey (SASS), "Public School Data File," 2007-08. 

Table 4 (see page 16) presents the descriptive statisties for the variables used in the 
analyses. Twice as many teachers partieipated in the spring survey (35) as did in the fall survey 
(18). The overall measure of level of implementation (“Aggregated Implementation”) was 
positively correlated with end-of-eourse aehievement (r=.08) and mean prior aehievement 
(r=.16). The positive correlation with mean prior achievement suggests that teaehers with higher- 
achieving students made greater use of QualityCore’s resources. The fall and spring measures of 
implementation were highly correlated (r=.68), suggesting that teachers’ level of implementation 
was consistent across the two semesters. Among the implementation domains, Collaboration 
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with Colleagues (r=.10) and Teaching Practices (r=.08) had the highest eorrelations with end-of- 
course aehievement, while QualityCore Adoption had no eorrelation (r=.00). Formative Items 
(r=.22) had the highest eorrelation with mean prior aehievement level, suggesting that teaehers 
were more likely to use QualityCore’s FIP with higher-aehieving students. The correlation 
among the five domains of implementation was lowest for the relationship between Teaching 
Practices and QualityCore Adoption (r=.03) and highest for Teaching Practices and Educator 
Resources (r=.76). 

The academic achievement data suggest that the students sampled were generally lower- 
achieving in comparison to students nationally. The mean QualityCore Algebra I score was 143 
(Table 4, see page 17); a score of 143 is at the 34th percentile nationally. The mean EXPLORE 
Mathematics score was 13.2; a score of 13 is at the 21st percentile nationally for students tested 
in fall of 9th grade (ACT, 2007). On average, the entering 9th grade students were over one 
standard deviation below the EXPLORE 8*'^ grade College Readiness Benchmark score of 17. 
The correlation of initial mathematics achievement level (EXPLORE Mathematics) and the 
spring EOC assessment score (QualityCore Algebra I) was 0.38. 

Among the five domains used to define the various aspects of implementation. 
Collaboration with Colleagues had the smallest mean rating of 3.0 — representing a slight 
disagreement on the level, or about 3 days a week, of collaborative work dedicated to the 
students, while Teaching Practices resulted in the highest mean rating of 4.5 (indicating slight- 
moderate agreement on statements of teaching practice, or a frequency of 4-5 days per week of 
various teaching practices). The standard deviations of the five domains ranged from 0.7 
{Teaching Practices) to 1.3 {Formative Items), suggesting that teachers did not vary much in 
their responses to the items assessing teaching practices, but varied more in their use of the EIP. 
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Hierarchical Modeling Results 

Relationship of level of implementation and student achievement. Table 5 contains 
the results of the hierarchical linear model assessing the relationship of level of implementation 
and student achievement in Algebra I. The aggregated implementation measure was significantly 
predictive of end-of-course student achievement, controlling for prior achievement (EXPLORE 
scores) and the group mean EXPLORE Mathematics score. A one point increase in the overall 
fall and spring implementation level was associated with a 0.66 point increase in the QualityCore 
Algebra I score (beta=0.66, SE=0.27, p-value=0.020). This result addresses the primary research 
question {To what extent did level of implementation of QualityCore Algebra I correspond to 
student achievement in Algebra 11). A one standard deviation increase in implementation 
corresponded to a 0.13^ standard deviation increase in end-of-course achievement level. 

Table 5 


Combined Fall and Spring Implementation Model Results 


Effect 

Estimate 

SE 

P-value 

Effect 

size 

Intercept 

142.86* 

0.18 

<0.0001 


Mean EXPLORE Math 

0.04 

0.12 

0.7227 

0.02 

EXPLORE Math 

0.21 

0.03 

<0.0001 

0.21 

EXPLORE English 

0.12 

0.04 

0.0008 

0.12* 

EXPLORE Reading 

0.05 

0.04 

0.1467 

0.05 

EXPLORE Science 

0.15 

0.04 

0.0001 

0.12 

Aggregated Implementation 

0.66 

0.27 

0.0201 

0.13’ 

Variance of intercepts 

0.97 

0.35 

0.0025 


Residual variance 

9.57 

0.38 

<0.0001 



N=l,291 students, 41 teachers; R = 0.50; *p-value < 0.05 


* The effect size estimate of 0.13 is derived as the parameter estimate associated with the aggregated implementation 
measure (0.66) multiplied by the standard deviation of the aggregated implementation measure (0.7), and divided by 
the standard deviation of the Algebra I EOC test score (3.5) (0.13 = 0.66*0.7/3.5). 
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Beyond the QualityCore implementation measure, there was other unexplained variation 
aeross teaehers in student performance on the EOC assessment: The variance of the teacher 
intercepts was estimated at 0.97 (p-value<0.01); this is evidence of variability in teacher effects 
on student achievement in Algebra I. Mean prior achievement level was not significantly 
predictive of EOC performance. As expected, EXPEORE math (beta=0.21, p-value<0.001) was 
predictive, and students’ prior achievement in English and science also helped predict EOC 
performance. 

A measure of intra-class correlation coefficient (Hedges & Rhoads, 2011) was computed 
as 0.09 (0.97 / [0.97 + 9.57]). Thus, there was 9% similarity in the QualityCore Algebra I 
achievement among students taught by the same teacher (with student variation in EXPEORE 
scores and mean EXPLORE math score accounted for). This can also be viewed as the 
percentage of total variation in the QualityCore Algebra I performance associated with 
differences between teachers (Singer, 1998). 

To get a sense of how much of the teacher variation was attributable to QualityCore 
implementation differences, we compared the fit of the model with and without the 
implementation variable. Erom the model without the implementation variable to that with it, the 
variance between teachers decreased from 1.06 to 0.97, a 0.09 decrease. Thus, although there 
was a significant relationship between level of implementation and achievement, it explained 
only 9% of the explainable variation in QualityCore Algebra I performance between teachers. 

Although the aggregate measure of implementation was significantly predictive of EOC 
performance, when broken down by semester, only the fall implementation measure had a 
significant relationship. A one-point increase in the fall implementation measure resulted in a 
0.94 point increase in QualityCore Algebra I score (Table 6, beta=0.94, p-value<0.05). In 
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contrast, the spring implementation measure was not signifieantly predietive of Algebra I seore, 
controlling for individual and group prior achievement (EXPLORE scores). The spring 
implementation effeet was estimated at 0.28 (Table 7), with a standard error of 0.27 (p- 
value>0.05). The results for the other predietor variables and varianee eomponents from the 
models testing the fall and spring implementation measures are similar to those observed in 
Table 5 for the aggregate implementation measure. 

Table 6 


Fall Implementation Model Results 


Effeet 

Estimate 

SE 

P-value 

Effect 

size 

Intereept 

143.00* 

0.27 

<0.0001 


Mean EXPLORE Math 

-0.13 

0.20 

0.5067 

-0.05 

EXPLORE Math 

0.23 

0.05 

<0.0001 

0.24 

EXPLORE English 

0.12 

0.06 

0.0669 

0.11 

EXPLORE Reading 

0.02 

0.06 

0.7504 

0.02 

EXPLORE Seienee 

0.12 

0.06 

0.0371 

0.11 

Eall Implementation 

0.94 

0.35 

0.0172 

0.19* 

Varianee of intercepts 

0.88 

0.52 

0.0455 


Residual varianee 

¥ 

9.18 

0.59 

<0.0001 



N=507 students, 18 teachers; R = 0.47; *p-value < 0.05 
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Table 7 


Spring Implementation Model Results 


Effect 

Estimate 

SE 

P-value 

Effect 

size 

Intercept 

142.86* 

0.18 

<0.0001 


Mean EXPLORE Math 

0.17 

0.12 

0.1578 

0.07 

EXPLORE Math 

* 

0.19 

0.04 

<0.0001 

0.18 

EXPLORE English 

0.10 

0.04 

0.0087 

0.10’ 

EXPLORE Reading 

0.10 

0.04 

0.0094 

0.09 

EXPLORE Science 

0.14 

0.04 

0.0006 

0.12* 

Spring Implementation 

0.28 

0.27 

0.3070 

0.05 

Variance of intercepts 

0.68 

0.29 

0.0092 


Residual variance 

9.58 

0.41 

<0.0001 



N=l,117 students, 35 teachers; R = 0.50; *p-value < 0.05 


Implementation domains and student achievement. To gain more insight into the 
relationship between implementation and achievement in student achievement, we tested 
whether the five implementation domain measures were predictive of student achievement. This 
analysis helps us understand which aspects of QualityCore implementation were most related to 
student achievement. We first entered each domain measure into the model individually (without 
the other four domain measures). These models are identical to the combined fall and spring 
implementation model (Table 5), except that each domain measure replaces the aggregate 
implementation measure. Then, we fit a model that included all five domain scores as predictors. 
The two sets of analyses let us examine each domain score’s relationship with student 
achievement, with and without consideration of scores from the other domains. 

Presented in Appendix B, Tables B-1 though B-5 contain the parameter estimates for 
each of the five implementation domains assessed individually. While the estimates for all five 
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domains are positive, only the estimates for Collaboration with Colleagues and Teaching 
Practices are significantly predictive of EOC performance, conditioning on individual and group 
EXPLORE scores. A one point increase in the Teaching Practices measure was associated with a 
0.52 point increase in the QualityCore Algebra I score (p-value<0.05). A one point increase in 
the Collaboration with Colleagues measure was associated with a 0.46 point increase in the 
QualityCore Algebra I score (p-value<0.05). The other domains showed positive but non- 
significant relationships. These results inform the second research question {Which aspects of 
QualityCore implementation (domains) were most related to student achievement in Algebra I?). 

Table 8 below contains the output of the hierarchical linear model of QualityCore 
Algebra I EOC scores regressed on all five implementation domain measures, controlling for 
individual and group EXPLORE Mathematics scores. When the implementation domain 
measures were considered jointly in the same model, only Teaching Practices was significantly 
predictive of EOC performance, conditioning on individual and group EXPLORE scores. A one 
point increase in the Teaching Practices measure was associated with a 0.65 point increase in the 
QualityCore Algebra I score (SE=0.29, p-value<0.05). Domains showing a positive but non- 
significant relationship include Collaboration with Colleagues (beta=0.24). Formative Items 
(beta=0.04), and QualityCore Adoption (beta=0.26). Educator Resources had a negative but non- 
significant relationship (beta=-0.37). 
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Table 8 


Domains of Implementation, Full Model Results 


Effeet 

Estimate 

SE 

P-value 

Intereept 

142.89* 

0.19 


Mean EXPLORE Math 

0.09 

0.13 

0.4943 

EXPLORE Math 

0.21 

0.03 

<0.0001 

EXPLORE English 

0.12 

0.04 

0.0009 

EXPLORE Reading 

0.05 

0.04 

0.1513 

EXPLORE Seience 

0.15 

0.04 

0.0002 

Collaborate with Colleagues 

0.24 

0.27 

0.3935 

Edueator Resourees 

-0.37 

0.30 

0.2359 

Eormative Items 

0.04 

0.30 

0.8908 

Teaehing Praetiees 

0.65 

0.29 

0.0319 

QualityCore Adoption 

0.26 

0.37 

0.4781 

Varianee of intereepts 

0.98 

0.37 


Residual varianee 

9.57 

0.38 



N=l,291 students, 41 teachers; R = 0.50; *p-value < 0.05 


Further assessment of the implementation domains model showed that transitioning from 
the model without to a model with the five domains of implementation, the between-teaehers 
varianee deereased from 1.04 to 0.98 (Table 8), a 0.06 point deerease. Thus, the five domain 
measures aeeounted for 6% of the explainable variation (Snidjers and Bosker, 1999) in 
QualityCore Algebra I performanee between teaehers. In addition, the intra-elass eorrelation was 
0.09 (0.98 / [0.98 + 9.57]), so 9% of the total variation in QualityCore Algebra I performanee 


was assoeiated with differenees between teaehers. 
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Survey Results Measuring QualityCore Use 

Beyond the investigation of QualityCore implementation’s relationship to student 
achievement, the survey results provided some insights into what components of QualityCore 
were used most by teachers. Additionally, the results provided information on teachers’ 
perceptions of QualityCore’s tools and overall effectiveness. As described earlier, the study 
included 41 teachers who had been surveyed and had students with EXPLORE and QualityCore 
Algebra I EOC data. In all, 56 teachers were surveyed — 15 teachers did not have students with 
EXPLORE or EOC assessment scores. In Appendix A, we summarize the survey responses for 
these 56 teachers. Across survey items, the maximum possible sample size was 69, as 13 of the 
56 teachers took both the fall and spring survey and we counted both sets of responses. We 
organized the results by implementation domain, and focused on the items that refer specifically 
to QualityCore tools and resources or the QualityCore program. 

Collaboration with Colleagues. The vast majority of teachers (90%) reported talking to 
colleagues about instructional strategies appropriate for specific QualityCore objectives (course 
standards). However, teachers only “slightly agreed,” on average, that teachers in their 
department used a common QualityCore terminology around teaching, assessment, and student 
work. Only about one-half of the teachers reported that they used QualityCore’s Worksheet to 
Examine Student Work to determine expectations for high-quality student work. 

Educator Resources. On average, teachers “moderately agreed” that they used 
QualityCore Depth of Knowledge Analysis (to adapt tasks to the needs of students), know how 
to use most of the resources in QualityCore’s Educator Toolbox, and know how to examine 
lessons for level of rigor and coherence with QualityCore course objectives. Teachers only 
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“slightly agreed” that they used QualityCore’s Template to Examine Assignments for Rigor & 
Relevanee. 

Formative Items. On average, teaehers reported using eonstrueted-response items from 
the FIP 2.8 days (out of 5 regular instruetional days). Similarly, they reported using multiple 
ehoiee items from the FIP 2.3 days. Teaehers “slightly agreed” that they frequently used items 
from the FIP during the year, and 21% agreed that they have not yet used items from the FIP to 
eonstruet elassroom-based assessments. Teaehers “slightly agreed,” on average, that they found 
opportune times to ineorporate items from the FIP in their elassroom. 

Teaching Practices. The majority of teaehers (81%) agreed that they eelebrated students’ 
progress towards QualityCore’s eourse standards by exhibiting student work in their elassroom. 
The vast majority (95 to 98%) of teaehers agreed with statements deseribing QualityCore- 
targeted teaehing practices, including; following a process for modifying lesson plans after 
examining student work, being able to describe how scaffolding was related to different depth of 
knowledge levels, and being able to describe the importance of a course syllabus in a rigorous 
high school course. 

QualityCorc Adoption. Nearly ah teachers (94%) reported that they had taken steps to 
design their curriculum so that it was aligned with the knowledge and skills necessary for 
college. On, average teachers slightly to moderately agreed (mean responses of 4.6) that they 
would recommend that other schools implement QualityCorc courses and that they were excited 
about implementing QualityCorc; teachers were slightly less likely to indicate that their 
colleagues were excited about implementing QualityCorc (mean response of 4.2). Most teachers 
(89%) agreed that they had observed positive results from QualityCorc, although the mean 
response (4.4) suggests that teachers tended to only “slightly agree.” Most teachers reported that 
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they had revised many of their lesson plans to intentionally include QualityCore (mean 
agreement response of 4.5) and that they often thought about how to incorporate the Model Unit 
into their instruction (mean of 4.6). About one quarter of the respondents agreed that they did not 
have the time to use QualityCore elements in their current work. 

Other survey items related to Adoption suggested less QualityCore buy-in from teachers. 
Only 58% of the teachers agreed that they planned on using the EOC assessment results to set a 
baseline for future years. Only 50% agreed that they had discussed (with colleagues) sections 
that needed to be added to the course to meet QualityCore course standards. Just over half of the 
teachers agreed that QualityCore did not align very well with the state’s performance standards; 
as discussed later this finding was confounded by state -mandated EOC assessments that were 
used for school accountability. 

Discussion 

Relationship of Implementation and Student Achievement 

We examined the relationship between QualityCore Algebra I level of implementation 
and student achievement. Eevel of implementation was measured by teachers’ self-reported 
utilization of QualityCore Algebra I resources and adherence to related practices. Achievement 
was measured by QualityCore Algebra I EOC scores controlling for prior EXPEORE scores. We 
found that, on average, an increase in level of implementation was significantly associated with 
increased performance on the QualityCore Algebra I EOC assessment. 

The study results are consistent with prior studies that have suggested that raising the 
achievement level of students before they enter high school is likely to be more effective in 
improving college readiness than other interventions (Sawyer, 2008). The estimated effect of 
level of implementation was 0.13 (meaning that a “large” increase in level of implementation 
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was related to a 0.13 standard deviation increase in EOC performance). To help put this estimate 
into perspective, a large (one standard deviation) increase in EXPLORE Mathematics score was 
related to a 0.38 standard deviation increase in EOC performance. Thus, the implementation 
effect size was about one-third the size of that for prior mathematics achievement. Erom this 
perspective, level of implementation appears to be an important contributor to student 
achievement. 

However, we cannot make a strong argument from this study that greater levels of 
implementation caused greater achievement. There could be unobserved variables related to both 
greater implementation and greater student achievement. In addition, as discussed later, there are 
limitations to the study that further temper the argument of a causal effect of implementation on 
student achievement. Euture research could strengthen the causal argument. Eor example, the 
QualityCore Algebra I EOC could be administered to students whose teachers did not participate 
in the study. A comparison of EOC scores with the scores of the students in this study, adjusted 
for prior achievement and other background characteristics, would be a stronger test of the 
QualityCore implementation effect. 

We also considered the fall and spring implementation measures separately. Although the 
aggregate fall and spring implementation was significantly predictive of student achievement, we 
found that only the fall implementation measure was significant in the term-specific models. One 
possible explanation for this result is that the fall survey included more items (49) than the spring 
survey (21) (see Appendix A), and thus should have been a more reliable measure of 
implementation. If, in fact, a relationship truly exists between level of QualityCore 
implementation and student achievement, we would be less likely to observe a statistical 
relationship with a less reliable measure of implementation. Another possible explanation is the 
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differences in fall and spring survey content. The fall survey included more items that were not 
directly related to the QualityCore course (and less likely to be attributed to QualityCore training 
and implementation) but likely connected to student achievement. Examples of such items 
included “I talk with teachers about student progress in grade levels other than those that I 
teach,” “Teachers in our department do not collaborate on instructional planning,” and “I have a 
hard time getting my students to take responsibility for their own learning.” It is possible that the 
fall survey better measured constructs related to student achievement, resulting in a significant 
effect on EOC student achievement of the fall and aggregated QualityCore implementation. 

Implementation was also measured for five domains, defined to measure different aspects 
of the tools, strategies, or practices of QualityCore Algebra I. A goal of the domain 
classifications was to be able to assess how each component contributed in raising the quality 
and intensity of the Algebra I course. While all domains had a positive relationship with student 
achievement, only Teaching Practices and Collaboration with Colleagues were statistically 
significant at the 0.05 significance level. The survey content for these two domains included 
more items that were not directly related to QualityCore resources, but perhaps more related to 
general teaching practices and positive school atmosphere (e.g., collaborative lesson planning) 
that were supported by QualityCore training and resources. It is possible that these general 
factors were more important than greater use of the specific resources offered by QualityCore 
(e.g., greater use of the EIP). 

QualityCore Use and Perceptions 

The survey results shed light on the extent that QualityCore resources were used, as well 
as teachers’ perceptions of QualityCore. Eor example, collaborative review of student work is a 
practice that is covered in QualityCore training and supported with QualityCore resources, but — 
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relative to other praetiees — was not reported as mueh by teaehers. On average, teaehers only 
slightly agreed that teaehers in their department believed that instruetion was improved by 
eollaboratively looking at student work. Only 61% agreed that they and their eolleagues came to 
a consensus as to what constituted high-quality student work, and only 54% agreed that they and 
their colleagues used the QualityCore Worksheet to Examine Student Work. 

In interpreting the survey results, it is important to keep in mind the timing of the survey 
(2009-2010 school year) and the standards, testing, and accountability systems in place at that 
time. In all three large school districts that were part of the study, 9th grade students were 
required to take the state’s EOC assessment in 9th grade Mathematics. In at least one district, the 
state EOC assessment accounted for a significant portion of students’ course grade. In all 
districts, the state EOC assessment was used for school accountability. Thus, the QualityCore 
EOC assessment was likely perceived as less important (by teachers and students) than the state 
EOC assessment. 

Also, 56% of the teachers agreed that QualityCore did not align very well with the state’s 
performance standards. In one of the study districts, the 9th grade math course was not called 
“Algebra I,” but resembled an integrated Mathematics course instead. In that district, the course 
standards did not match well with those of traditional Algebra I courses. Accordingly, 67% of 
the teachers agreed that QualityCore Algebra I did not align very well with the state’s 
performance standards. In comparison, in the other two districts that did have an Algebra I 
course for 9th graders, only 22% of the teachers agreed that QualityCore Algebra I did not align 
very well with the state’s performance standards. Because most states have since adopted the 
Common Core State Standards and because QualityCore’s course standards align well with the 
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Common Core State Standards (ACT, 2010b), it is likely that current and future studies of 
QualityCore implementation would suggest stronger alignment. 

Study Limitations 

The study has some important limitations that should be acknowledged. First, the number 
of items used for the fall survey greatly outnumbered those used in the spring, with only 13 items 
used for both the fall and spring surveys. The two surveys differed in both content and reliability, 
and cannot be considered repeated measurements of the same implementation construct. Ideally, 
the same survey would have been administered during both terms so that reliability would be 
more uniform and better assessed. Moreover, using the same survey in both terms would have 
provided a better way to measure change in implementation across the year. 

Another limitation is that more teachers were surveyed in the spring than in the fall. A 
more accurate measure of the relationship between implementation and achievement could have 
been achieved with the same teachers surveyed in fall and spring. For future studies, we would 
recommend a unified and reconstructed questionnaire, building off of the items provided in 
Appendix A. Only 41 teachers were included in the primary analyses. Because the effect of any 
one -year program or intervention (including QualityCore Algebra I) on student achievement is 
likely to be small, a larger sample size of teachers and students is needed in order to draw more 
conclusive results and to have adequate power to detect effects of different aspects of 
implementation. 

As noted above, the study was conducted in three large school districts with high 
percentages of lower-achieving students, many living in poverty. The adoption of QualityCore 
was funded by school improvement grants, and was part of a comprehensive strategy for 
improvement in the three districts. The use of QualityCore, and the effect of QualityCore on 
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Student achievement in Algebra I, could both be influenced by these contextual factors, and limit 
the generalization of the findings. The vast majority of students (85%, see Table 3) in the sample 
were African American, which could also affect the generalization of the findings. 

The measurement of implementation relied on teachers’ self-report. With self-report, 
social desirability bias is a concern; teachers have been shown to over-report positive changes in 
classroom practices, which is attributed to a desire to appear favorably in relation to one’s peers 
(Kopcha & Sullivan, 2006). Teachers were told in the survey instructions that their responses 
would remain anonymous, which can help reduce this bias (Scaeffer, 2000). Still, it is possible 
that teachers over-reported their use of QualityCore resources and teaching practices, and this 
may have weakened the observed relationship between level of implementation and student 
achievement. 

In addition to the reliance on self-reported levels of implementation, there is also a 
concern that survey non-response could have affected the results. We estimate that 37% of the 
teachers implementing QualityCore Algebra I participated in one or more surveys; it is possible 
that the non-participants had different levels of implementation and that the relationship between 
implementation and student achievement was different for the non-participants. If the 
relationship between level of implementation and student achievement was weaker for the non- 
participants, the survey non-response bias would result in this study reporting an inflated 
relationship between level of implementation and student achievement. 

The study assessed the relationship of level of QualityCore Algebra I implementation and 
student achievement. QualityCore implementation was conceptualized as a “treatment” measured 
on a scale; the study did not, however, directly test whether QualityCore adoption was related to 
improved student achievement (e.g., QualityCore adoption versus no use of QualityCore). The 
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Study found evidence of a positive relationship between level of implementation and student 
achievement, but future studies should examine the impact of QualityCore adoption on students’ 
college and career readiness. A recent large-scale study has documented positive improvements 
in college readiness associated with QualityCore adoption, with larger improvements observed 
for schools that participated in professional development on formative instructional practices 
(Battelle for Kids, 2012). Future research should continue to examine the effects of QualityCore 
adoption and level of implementation on school-wide achievement and growth. 
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Over the past 5 regular instructional days, how often have you performed each of the following activities? 

0=No time, 1=1 hour, 2=2 hours, 3=3 hours, 4=4 hours, 5=5 hours, 8=6-10 hours, 15=1 1 hours or more 

Q 

cc 


On 

Please rate the level of your agreement with each of the following statements. 

l=Strongly Disagree, 2=Moderately Disagree, 3=Slightly Disagree, 4=Slightly Agree, 5=Moderately Agree, 6=Strongly Agree, Not Applicable 
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Fall 

X 

X 

X 

X 

X 

X 

X 

Item 

Collaboratively "Looking at Student Work" 

Collaboratively planning lessons 

Item 

My colleagues and I talk about instructional strategies that would be 
appropriate for teaehing specific QualityCore objectives. 

I talk with teachers about student progress in grade levels other than 
those that I teaeh. 

My eolleagues and I have trouble deciding on ways to change our 
instruction based on collaborative review of student work. [R] 

I have discussed my course syllabus with other teachers who teach the 
same eourse. 

In my department, teaehers believe that instruction for all students is 
improved by collaboratively looking at student work. 

I rarely talk to teachers in departments other than Math about course 
objectives and students progress. [R] 

Teachers in my department use a common QualityCore terminology to 
deseribe teaching, assessment, and student work. 





Collaboration with Colleagues 


oo 


Please rate the level of your agreement with each of the following statements. 

l=Strongly Disagree, 2=Moderately Disagree, 3=Slightly Disagree, 4=Slightly Agree, 5=Moderately Agree, 6=Strongly Agree, Not Applicable 
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18 

My colleagues and I have a scheduled time when we meet to plan 
instruetion and analyze student work. 

Teachers in our department do not collaborate on instruetional 
planning. [R] 

Item 

My colleagues have asked me about the results I have observed using 
new practices I learned in QualityCore training. 

At faculty meetings, we regularly discuss student successes. 

My colleagues and I do not follow a process when examining student 
work. [R] 

My colleagues and I come to a consensus as to what constitutes high- 
quality student work. 

Teachers in my department are not given much time to collaboratively 
plan. [R] 

At my school, experienced faculty mentor to less experienced faculty. 

My colleagues and I use a QualityCore Worksheet to Examine Student 
Work and determine expectations for high-quality student work. 

Number of items 





o^ 

m 


Please rate the level of your agreement with each of the following statements. 

l=Strongly Disagree, 2=Moderately Disagree, 3=Slightly Disagree, 4=Slightly Agree, 5=Moderately Agree, 6=Strongly Agree, Not Applicable 
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Over the past 5 regular instructional days, how often have you used the foil 
Educator’s Toolbox? 

5=Daily (5 days), 4=Frequently (4 days), 3=Often (3 days), 2=Sometimes (2 days), l=Seldom 

Fall 

X 

•X- 
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•X- 

X 

•X- 

X 

•X- 

X 

•X- 

X 

•X- 

* 

Item 

I use QualityCore Depth of Knowledge analysis to adapt tasks to the 
needs of my students. 

I use QualityCore ’s Template to Examine Assignments for Rigor and 
Relevance to help me modify assignments. 

I know how to use most of the resources in the QualityCore Educator’s 
Toolbox. 

I know how to examine lessons for level of rigor and coherence with 
QualityCore Course Objectives. 

Item 

Formative Assessment (e.g., Index Cards, K-W-E, Clickers, 3-2-1 
Assessment, Traffic Eight Icons) 

Problem-Based Inquiry (e.g.. Systems Analysis, Essential Questions, 
Question Wall) 

Collaborative Discourse (e.g.. Whiteboard Activities, Dialogic 
Journals, Group Work, Four Comers ) 

Assessing Prior Knowledge (e.g.. Background Knowledge Probe, 
Entrance and Exit Slips) 

Practice, Review, and Revision (e.g.. Focused Fisting, Three-Minute 
Review, Advance Organizer) 

Sense-Making and Wrap-up (e.g.. Muddiest Point, Minute Paper, 
Journals and Teaming Fogs) 

Number of items 











Teaching Practices 


Over the past 5 regular instructional days, how much time have you spent on the following activities? 

5=Daily (5 days), 4=Frequently (4 days), 3=Often (3 days), 2=Sometimes (2 days), l=Seldom (1 day), 0=Never (0 days). Not Applicable 

Q 

IZJ 

os 

o 

- 


- 


Over the past 5 regular instructional days, how much time have you spent on the following activities? 

0=No time, 1=1 hour, 2=2 hours, 3=3 hours, 4=4 hours, 5=5 hours, 8=6-10 hours, 15=1 1 hours or more 

Q 

cc 

(N 

(N 

(N 

rn 

Please rate the level of your agreement with each of the following statements. 

l=Strongly Disagree, 2=Moderately Disagree, 3=Slightly Disagree, 4=Slightly Agree, 5=Moderately Agree, 6=Strongly Agree, Not Applicable 

Q 
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(N 
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Al 
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%Agree 
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<N 

(N 

CO 

Z 
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(N 

Spring 

X 

X 



X 

Spring 


X 

Spring 




Fall 

X 

X 

X 

X 

X 

Fall 

X 

X 

Fall 

X 

X 

X 

Item 

I provided my students with a warm-up (or hook) to review previous 
lessons or to antieipate a new lesson. 

I provided rationale to my students regarding the focus and purpose of 
the day’s lesson. 

I used scaffolding during the class period. 

I concluded my lesson with a summary. 

I instructed my students to summarize their learning with a wrap-up 
activity (closing, exit slip, journal, etc.) 

Item 

Independently grading assignments/tests and "Looking at Student 
Work" 

Independently planning lessons 

Item 

I celebrate students’ progress towards the QualityCore course standards 
by exhibiting student work in my classroom. 

My course syllabus does not include a personal statement. [R] 

I often adapt assignments to ensure that they have sufficient rigor and 
relevance 









QualityCore Adoption 


Please rate the level of your agreement with each of the following statements. 

l=Strongly Disagree, 2=Moderately Disagree, 3=Slightly Disagree, 4=Slightly Agree, 5=Moderately Agree, 6=Strongly Agree, Not Applicable 

as 



(N 

lO 

0.9 

OO 

(N 




On 



4.6 

4.5 

3.7 

2.5 

5.2 

3.7 

4.4 

4.2 

4.6 

4.6 

3.9 


%Agree 

80 

78 

58 

26 

94 

50 

89 

75 

94 

OO 

56 


Z 

20 

09 

52 

os 

OO 

OO 

OO 

20 

OO 

20 

39 
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X 
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X 


Fall 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 


10 

Item 

I often think about how I can incorporate features of the QualityCore 
Model Instructional Unit into my instruction. 

I have revised many of my existing lesson plans to intentionally 
include QualityCore resources into my teaching. 

I plan on using this year’s QualityCore end-of-course exam results to 
set a baseline for future years’ student achievement. 

I don’t have the time right now to use QualityCore elements in my 
work. [R] 

I have taken steps to design my curriculum so that it is aligned with the 
knowledge and skills necessary for college. 

My colleagues and I have discussed sections that need to be added to 
the course to meet the standards of QualityCore objectives. 

I have observed positive results from implementing QualityCore in my 
classroom. 

My colleagues are excited about implementing QualityCore in the 
classroom. 

I would recommend that other schools implement QualityCore courses. 

I am excited about implementing QualityCore in the classroom. 

QualityCore doesn’t align very well with our state’s performance 
standards. [R] 

Number of items 


R indicates reverse-score item. 





Item Fall Spring N % Agree 

Number of items 49 21 
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Appendix B 


Results of Hierarchical Linear Models 
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Table B-1 


Partial model results: Collaboration with Colleagues 


Effect 

Estimate 

SE 

P-value 

Effect 

Size 

Intercept 

142.91* 

0.19 

<0.0001 


Mean EXPLORE Math 

0.02 

0.12 

0.8357 

0.01 

EXPLORE Math 

0.21 

0.03 

<0.0001 

* 

0.21 

EXPLORE English 

0.12 

0.04 

0.0012 

* 

0.11 

EXPLORE Reading 

0.05 

0.04 

0.1595 

0.05 

EXPLORE Science 

0.15 

0.04 

0.0002 

0.12 

Collaboration with Colleagues 

0.46 

0.21 

0.0375 

0.11 

Variance of intercepts 

0.92 

0.35 

0.0043 


Residual variance 

9.60 

0.39 

<0.0001 


*p-value < 0.05 





Table B-2 





Partial model results: Educator Resources 




Effect 

Estimate 

SE 

P-value 

Effect 

Size 

Intercept 

142.83* 

0.19 

<0.0001 


Mean EXPLORE Math 

0.06 

0.12 

0.6412 

0.02 

EXPLORE Math 

0.21 

0.03 

<0.0001 

0.21* 

EXPLORE English 

0.10 

0.04 

0.0009 

0.12 

EXPLORE Reading 

0.05 

0.04 

0.1544 

0.05 

EXPLORE Science 

0.15 

0.04 

0.0001 

0.12 

Educator Resources 

0.30 

0.18 

0.1191 

0.09 

Variance of intercepts 

1.04 

0.38 

0.0028 


Residual variance 

9.59 

0.39 

<0.0001 



*p-value < 0.05 
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Table B-3 

Partial model results: Formative Items 


Effect 

Estimate 

SE 

P-value 

Effect 

Size 

Intercept 

142.84’ 

0.19 

<0.0001 


Mean EXPLORE Math 

0.05 

0.12 

0.7051 

0.02 

EXPLORE Math 

0.21 

0.03 

<0.0001 

0.21 

EXPLORE English 

0.12 

0.04 

0.0011 

0.11* 

EXPLORE Reading 

0.05 

0.04 

0.1471 

0.05 

EXPLORE Science 

* 

0.15 

0.04 

0.0001 

0.12 

Eormative Items 

0.26 

0.21 

0.2153 

0.07 

Variance of intercepts 

1.09 

0.39 

0.0023 


Residual variance 

* 

9.59 

0.39 

<0.0001 


*p-value < 0.05 





Table B-4 





Partial model results: Teaching Practices 




Effect 

Estimate 

SE 

P-value 

Effect 

Size 

Intercept 

142.85’ 

0.18 

<0.0001 


Mean EXPLORE Math 

0.10 

0.11 

0.3852 

0.04 

EXPLORE Math 

0.21 

0.03 

<0.0001 

0.21 

EXPLORE English 

0.12 

0.04 

0.0007 

0.12 

EXPLORE Reading 

0.05 

0.04 

0.1510 

0.05 

EXPLORE Science 

0.15 

0.04 

0.0002 

0.12 

Teaching Practices 

0.52 

0.18 

0.0057 

0.15 

Variance of intercepts 

0.86 

0.32 

0.0037 


Residual variance 

9.58 

0.39 

<0.0001 



*p-value < 0.05 
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Table B-5 


Partial model results: QualityCore Adoption 


Effect 

Estimate 

SE 

P-value 

Effect 

Size 

Intercept 

142.8r 

0.20 


-0.04 

Mean EXPLORE Math 

0.06 

0.12 

0.6489 

0.02 

EXPLORE Math 

0.21 

0.03 

<0.0001 

0.21 

EXPLORE English 

0.12 

0.04 

0.00010 

0.12 

EXPLORE Reading 

0.05 

0.04 

0.1313 

0.05 

EXPLORE Science 

0.15 

0.04 

0.0001 

0.12 

QualityCore Adoption 

0.22 

0.25 

0.3808 

0.05 

Variance of intercepts 

1.10 

0.39 


0.09 

Residual variance 

9.59 

0.39 


0.77 


*p-value < 0.05 
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